Skip to content

Conversation

@KesterTan
Copy link

@KesterTan KesterTan commented Jan 26, 2025

Attempts to fix the copy-in issue that we're still facing even after retries.

  1. Captures invalid inputfiles structure
  2. Adds -p flag and permissions when creating autolab directory. This ensures that the specified directory and any necessary parent directories are created. If the directory already exists, the command does not raise an error, it would originally throw an error.
  3. Added thread pool to limit scp commands so that instance is not overloaded
  4. Added log statements with job id for more in-depth logging.

Ran a thousand jobs on https://dev.autolabproject.com/courses/test-course/jobs?id=500 without failure.

@KesterTan KesterTan marked this pull request as ready for review February 2, 2025 17:04
@KesterTan KesterTan requested a review from evanyeyeye February 2, 2025 17:04
@KesterTan KesterTan removed the request for review from evanyeyeye September 16, 2025 16:48
@KesterTan KesterTan requested review from a team, 20wildmanj, anthony-yip and dwang3851 and removed request for a team and 20wildmanj September 16, 2025 19:34
Copy link

@anthony-yip anthony-yip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments, but a lot are just nitpicks. Talk to me how you want to proceed. I've also fixed some of these in my own PRs. Also, you mentioned thread pool but I'm not sure which one you're referring to?

Copy link

@dwang3851 dwang3851 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, left some small comments

Copy link

@anthony-yip anthony-yip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left 2 more comments, the ones I left last time were addressed

@KesterTan KesterTan requested a review from anthony-yip October 21, 2025 03:27
@KesterTan KesterTan merged commit 61e5d64 into ec2-new-implementation Oct 30, 2025
1 check passed
@KesterTan KesterTan deleted the copy-in branch October 30, 2025 21:57
@KesterTan KesterTan restored the copy-in branch November 11, 2025 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants